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The RICIS Concept 


The University of Houston -Clear Lake established the Research Institute for 
Computing and Information Systems (RICIS) in 1986 to encourage the NASA 
Johnson Space Center (JSC) and local industry to actively support research 
in the computing and information sciences. As part of this endeavor, UHCL 
proposed a partnership with JSC to jointly define and manage an integrated 
program of research in advanced data processing technology needed for JSC's 
main missions, including administrative, engineering and science responsi- 
bilities. JSC agreed and entered into a continuing cooperative agreement 
with UHCL beginning in May 1 986, to jointly plan and execute such research 
through RICIS. Additionally, under Cooperative Agreement NCC 9-16, 
computing and educational facilities are shared by the two institutions to 
conduct the research. 


The UHCL/RICIS mission is to conduct, coordinate, and disseminate research 
and professional level education in computing and information systems to 
serve the needs of the government, industry, community and academia. 
RICIS combines resources of UHCL and its gateway affiliates to research and 
develop materials, prototypes and publications on topics of mutual interest 
to its sponsors and researchers. Within UHCL, the mission is being 
implemented through interdisciplinary involvement of faculty and students 
from each of the four schools: Business and. Public Administration, Educa- 
tion, Human Sciences and Humanities, and Natural and Applied Sciences. 
RICIS also collaborates with industry in a companion program. This program 
is focused on serving the research and advanced development needs of 
industry. 


Moreover, UHCL established relationships with other universities and re- 
search organizations, having common lesearch interests, to provide addi- 
tional sources of expertise to conduct needed research. For example, UHCL 
has entered into a special partnership with Texas A&M University to help 
oversee RICIS research an i education programs, while other research 
organizations are involved via the “gateway” concept 

A major role of RICIS then Is to find the best match of sponsors, researchers 
and research objectives to advance knowledge in the computing and informa- 
tion sciences. RICIS, working jointly with its sponsors, advises on research 
needs, recommends principals for conducting the research, provides tech- 
nical and administrative support to coordinate the research and integrates 
technical results into the goals of UHCL, NASA/JSC and industry. 
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Executive Summary 


The Information Systems Directorate at JSC/NASA has undertaken 
the reevaluation of its performance measures process and measures. 
Under the direction of a quality approach it is essential to 
identify an external perspective of how well an organization is 
performing. This study was conducted with two major objectives: 

1 , Survey and summarize the academic literature on 
performance measures as indicators of information 
systems (IS) effectiveness^ 

2 , Survey organizations for their experience in measuring 
for IS effectiveness. 

Four approaches to measuring the effectiveness of IS 
performance were identified: 

1. Listen to the customer for the things they need 

2. Align with corporate goals 

3. Benchmark against well-respected organizations 

4. Ask yourself what critical factors lead to success 

The list of known methods for soliciting customer feedback 
are : \ 


Executive visit 
Survey, interview, focus group 
Complaints and compliments 
Service level agreements 

A common set of characteristics that satisfy customers was 
identified from the literature. The list includes elements such as 

Accuracy Understandability 

Timeliness Reliability 

Relevance Completeness 

Future research in this topic area should prove beneficial to 
determine the metrics for external validity. Suggested topics are 
listed at the end of this report. 

i 
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Exploratory Study on Performance Measures 
as Indicators of IS Effectiveness 


Introduction 

The quality approach being developed at NASA/JSC requires 
that each unit measure its performance in order to guide and 
document its improvement. The Information Systems Division (ISD) 
has been collecting measures of its performance for a number of 
years. These measures include system and network availability, 
response time, time to resolve problems, etc. These indicators 
are generally classified as productivity or efficiency measures 
because they document how much output is produced with a given 
amount of input. Productivity measures are common and well 
established in the field of information processing. 

Another class of measures is not as readily available. These 
are measures that relate to h ow well the organization is doing 
from an external perspective. The difference between these 
latter measures and productivity measures is captured in the 
phrase: 

"Doing things right" 
vs 

"Doing the right things" 

The former refers to productivity — whatever an organization 
does, it does well. The second questions whether the things it 
does are in fact the right things to do — activities that 
provide value to the enterprise. This study is designed to 
address the latter domain. The question is simply 

"How does ISD know when it is doing the right things?" 

Background 

The purpose of an exploratory study is to quickly and 
concisely provide an overview in a designated topic area. The 
focus of this research is to determine the availability of useful 


l 



information on performance measures as indicators of IS 
effectiveness. A quick survey and summary of the type of 
information available will be useful in determining if further 
research in this area will help ISD in developing meaningful 
performance measures. 

Objectives 

1. Survey and summarize the academic literature on 
performance measures as indicators of IS effectiveness. 

2. Survey organizations for their experience in measuring 
for IS effectiveness. 


Scope 


This study was conducted as a quick, exploratory search for 
available information on how to measure for IS effectiveness. 
Information gathered was confined to literature citations 
available through electronic searches and the library of the 
American Productivity and Quality Center, and through short 
surveys completed by IS managers in select local corporations. 

Approach 

Three basic sources of information were used in this study. 
First, a literature search of business databases was conducted 
using Dialog Information Services. The search terms were: 

(OUTCOME OR EVALUATION OR PERFORMANCE) AND 
(INDICATOR OR MEASURE) AND 

(INFORMATION SYSTEM OR DATA CENTER OR MIS) 

Most of the citations came from ABI/ INFORM, a database that 
covers all major business publications. 

A second source was the library at the American Productivity 
and Quality Center (AP&QC) of which NASA/ JSC is a member. The 
AP&QC holds an excellent collection of articles on all aspects of 
quality processes. 
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Finally, information was gleaned from actual company 
experience. NASA/ JSC is a member of the Information Systems 
Research Center (ISRC) at the University of Houston. The ISRC is 
comprised of 20 Major corporations in the Houston area. 
Information systems (IS) managers from member corporations were 
queried on how they measure their department's effectiveness. 

Data were collected in a 30-minute open-ended telephone interview 
or via a two-page, six question, faxed survey (Appendix A) . Six 
corporations participated in the survey. This is a 33% response 
rate which is considered very good. Companies responding 
included three oil and gas companies, a public utility, an 
environmental company and an insurance/ investments firm. 

Findings 

The value of information technology to modern organizations 
is widely accepted. Computers and telecommunications are playing 
and will play an important role in organizational effectiveness 
and success. Documenting that claim, however, has turned out to 
be difficult. Because of the intense interest in this subject, 
there is no lack of literature on the subject. The Dialog search 
turned up almost 600 references using the terms listed above in 
just the last three years. As a result, the actual citations 
used were limited to those in which those terms figured 
prominently in the title or the abstract. 

The overwhelming theme from all of these studies is that the 
IS customer is the one who ultimately decides whether IS "is 
doing the right things." In contrast to productivity measures of 
machine performance, IS cannot answer this question by itself. 

It requires information based on a partnership with the customer. 

A second and equally important theme is that such customer 
data by itself does not automatically result in a more effective 
organization. The results of such studies must be fed into an 
organizational process that uses the results for improvement. 

Types of Data 

The literature revealed four basic types of data that can be 
used to answer the question of external performance: customer 
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feedback, corporate alignment, benchmarking, critical success 
factors. 

1. CUSTOMER FEEDBACK 

The overwhelming majority of studies depended on IS 
customers to tell whether or not IS was offering the right 
products and services. Historically, IS organizations have 
always been customer-oriented. On the other hand, given the 
complexity of the technology, customers were usually in a poor 
position to know which technology was available in a cost- 
effective manner. As a result, IS organizations have typically 
"told" customers what they needed rather than asked them. With 
quality initiatives, that behavior has begun to change. 

The issue of the customer expectations is still a problem, 
since the media is continuously advertising the benefits of the 
next wave of technology (along with how inexpensive it is 
supposed to be) (Caravella, 1989). Asking customers what they 
want is a perennial problem in all businesses. The problem, 
however, does not remove the absolute necessity of getting the 
customer rather than the IS organization to decide what is right 
for them. As discussed below, this customer data comes in 
various forms. 

2 . CORPORATE ALIGNMENT 

The second type of data is a nebulous concept of corporate 
alignment. Simply put, IS doing the right thing when it supports 
the corporate strategy (Rubin, 1991) . Alignment is a tricky 
concept to measure because it involves interpretation of the 
details of the corporate strategy and just how IS linked to it. 
Rubin uses a model of alignment which breaks that concept into 
component parts. A systematic discussion within ISD of corporate 
goals and ISD's contribution to them should increase the 
alignment. 
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3 . BENCHMARKING 


Another type of secondary data is the emerging strategy of 
benchmarking — comparing one's organization with what is thought 
to be the best in the field. Doing the right things in this 
context means doing the same things which respected organizations 
do. Computerwor Id identified the Premier 100 IS organizations in 
September 1991 and intervi ewe d them for what they were doing: 

1. Doing more than one thing rather than betting on one big 
project. 

2. Looking for ways to hold off spending and getting the 
most from existing resources (although 58 companies are 
expecting increased budget) . 

3. Spending more money on end-user computing. 

4. Increasing their computer security. 

5. Other related trends: Re-engineering 

Downsizing 
Open systems 
Networks 

Customer service 
Global expansion 
Outsourcing 

Surprisingly only 65% of these best companies had "programs in 
place to measure the effectiveness of their IS operations." 
Likewise, Rubin (1991) reports that only 20% of 300 measurement 
programs begun in the 1980s were successes — that is, the 
program lasted more than 2 years and results were communicated 
and used. Thus it seems that measurement programs are much 
harder to implement than they appear. 
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Actual 

Value 


Expected 

Value 


Exhibit 1 


Rubin, 1991, 


p.3 


4. CRITICAL SUCCESS FACTORS (CSF) 

A fourth source of data on external performance is the 
investigation of the Critical Success Factors. CSFs are what the 
organization thinks it must do well to succeed. J.F. Rockart, a 
Harvard professor, applied the concept of CSF to IS organizations 
in a 1979 article. That article spawned a wave of studies 
looking into the CSFs for IS organizations. Appendix B contains 
the CSFs reported in these studies. They form a list of what 
other organizations have decided are their critical task areas. 

On the other hand, the CSF approach will probably be less 
popular in the 1990s because it has little or no external 
component. While analyzing what the organization thinks would be 
necessary for its success is commendable, the CSF approach has no 
external validation, such as customers, corporate goals, or 
benchmark organizations, to keep it honest. CSFs do provide a 
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functional breakdown of the critical task areas that an 
organization thinks are most important, but they are the least 
valid in answering the question about whether those are in fact 
the critical areas. 


Discussion 

Performance measures nourishes the planning cycle which in 
turn drives budgeting and implementation, as illustrated in 
Figure 1. The implementation is then judged either effective or 
not by the customer touching off another round of improvement. 
Planning is also informed by info rma tion about the organization 
itself and about the IS environment as it develops. 



Without such a process in place, performance measures may be 
interesting but they themselves will not lead to greater 
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effectiveness. With such a process in place, the whole character 
of management changes. In place of traditional management comes 
"Management by Fact," a concept derived from the 1992 Malcolm 
Baldridge National Quality Award Criteria (pp.3-4) . 

Meeting quality and performance goals of the 
company requires that process management be 
based on upon reliable information, data, and 
analysis. .. Facts, data, and analysis support 
a variety of company purposes, such as 
planning, reviewing company performance, 
improving operations, and comparing company 
quality performance with competitors. 

Just as we would no longer expect pilots to fly by the "seats of 
their pants," so in the future managers will not be able to 
direct their companies using facts. The companies and agencies 
which adopt this approach now will be more effective and 
competitive in the future. 

Characteristics of Effective IS Organizations 

An organization's activities bear on its performance, but a 
more direct approach is to measure the quality of the products 
and services it produces rather than just what it does. In this 
vein, many studies attempt to determine the characteristics of 
effective IS products and services in order to provide measures 
of what IS organizations should measure about their products and 
services. The list in Appendices 6 and C report the results. 

Most of these studies asked IS customers what made them most 
satisfied with IS systems and products. One comprehensive study 
(Bailey & Pearson 1983) asked 32 managers (IS customers) from 
eight organizations to describe what was most important to them 
about IS p rodu cts and services. Their responses were compared 
with a list of 38 items were extracted from the literature by 
asking the managers to rank the importance of each of the items. 
Of the top ranked 25 items, 14 items described systems and 
products and 11 described IS staff and services. These items are 
presented in Table I. Since they include the most important 
items from the rest of the other literature in this survey, they 
represent the best single list of the characteristics of 
effective IS organizations. Such a list can be used directly as 
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a set of external performance 
measures of a typical IS 
organization. 

These results reemphasize 
the new yardstick for 
performance — the ever- 
present and demanding 
customer. That customer who, 
as little as he or she knows 
about information technology, 
does know their own business 
better than the IS 
organization does and 
therefore can tell best how to 
improve it and make it 
effective. Successful IS 
organizations forge 
partnerships with these 
customers, offering them cost- 
effective products and 
services that enhance their 
ability to meet their goals 
and be successful. Thus, the 
simplest answer to the 
question "Are we doing the 
right thing?" is "Ask the 
customer . " 


Table I 

Product and Service 
Characteristics 


Product/ System 
Characteristics 


Accessible/convenient 
Accurate/ correct 
Complete 

Confident in the system 

Current/up-to-date 

Fast /responsive 

Flexible/adaptable 

Precise 

Recoverable 

Relevant 

Reliable 

Timely 

Understandable 

Useful 


Staff /Service 
Characteristics 

Attitude 
Communications 
Convenience 
Development time 
Document at i on 
Flexibility 
Participation/ control 
Priorities 

Request processing time 
Technical competence 
Training 


Techniques for measuring customer satisfaction 

A final class of literature dealt with the various ways that 
firms used to gauge their customer's satisfaction. Customer 
satisfaction is not directly related to whether it is actually 
"doing the right things." It focuses more on how well it does 
the things which it is currently doing. But customer studies can 
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also include an "importance" rating along with the satisfaction 
measures to gauge how critical a product or service is. 


1. customer feedback 

The literature contains four distinct methods for soliciting 
customer feedback, the most popular of which is annual customer 
survey. Surveys are ubiquitous today and will probably become 
even more popular in the future. As they spread, however, the 
average quality of surveys may actually decline. 

The most popular format for the items on customer surveys is 
the semantic differential scale. The scale describes the 
characteristic to be measured (e.g., degree of EDP training 
provided) and asks the respondent to place a mark between polar 
opposites which describe that item (e.g., complete or incomplete, 
sufficient or insufficient, etc.). The distance between the mark 
and the more positive pole is a measure of the customer's 
satisfaction on that item. 

Baroudi and Orlikowksi (1988) use the semantic differential 
in a simple 13-item instrument. The items were taken from a much 
longer list generated by Ives (1983), Each item is rated using 
two pair of polar opposites for a total of 26 scores. All the 
items except the convenience and flexibility of services also 
appear on Bailey and Pearson's lists in Table I. 

This approach is a straightforward method for measuring 
customer satisfaction on a pre-defined set of criteria. Also 
popular are the more common Likert scales in which each item on a 
5- or 7-point scale is labeled individually from "very satisfied" 
to "very dissatisfied." 

Goodman (1990) provides another alternative for customers 
who actually purchase IS services. He prefers to ask about some 
future action, such as "Will you use this product or service 
again?" or "Would you recommend this product or service to 
someone else?" 

Goodman also warns that "good" or "OK" may not be as 
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positive as it sounds. His study of banks showed that between 
20% and 50% of customers who said that a bank's performance was 
"good” switched banks at the next opportunity. Counting only the 
"above average" or "excellent" categories as successes will 
remedy that problem. 

Customer opinion can also be gathered in a number of other 
ways and at a variety of other times: 

Focus group (discussion groups on IS products/ services) 

Follow-up survey after product or service delivery 

Critical incident interview (Bitner, 1990) 

Pre- and post- implementation audit (Eckerson, 1991) 

2 . Executive Visits 

Another critically important method for measuring customer 
satisfaction is the executive visit (Cannella 1991; Goodman 
1982). Here the executive in charge of the IS organizations 
meets with the executive of a customer organization to solicit 
feedback and form an alliance for the future. The executive 
visit is crucial because it provides direct feedback to the 
person in charge of IS. It also shows IS's willingness to engage 
in discussion of its performance at the highest level. Finally, 
it allows both executives to agree on the major aspects of 
providing information technology to the customer's organization 
for the coming year. 

Both organizations can then respond to the same set of goals 
and expectations as they work together at the operational levels. 
An agreement on goals and strategies can also help manage the 
customer organization's expectations of what IS can perform in a 
year. At the same time, it places an extra burden on IS to 
deliver what it committed to. The executive interview is an 
excellent addition to any performance measurement program. 


3. Complaints and Compliments 

A third source of customer satisfaction data is unsolicited 
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complaints and compliments. Goodman (1982) argues that companies 
should make it easy for customers to complain. They might even 
make them feel good about it! The reasons that people do not 
complain are fairly obvious: 

— People do not believe that complaining will do any good. 

— People assume that poor service comes with a low price. 

The person who receives the poor service is too busy. 

— The person who receives the poor service is not the 
person who makes the purchase decision. 

— The person does complain to the service representative 

but the representative cannot resolve the complaint and 
does not report it. 

Companies who believe that complaints are important for the 
quality of their products and service take positive steps to 
increase the number of complaints: 

— Constant top management contact with customers 

— Rapid response to problems 

— Fewer obstacles to complaining 

— Frequent surveys to solicit complaints 

— Internal incentives to find and resolve complaints 

Goodman concludes with an eight item self-evaluation 
checklist on how well a company can use complaints as a major 
source of customer satisfaction data (Goodman 1982, p.44). 
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I. Do your customers know how and where to ask or complain? 

II. If a customer has a repeat problem, is there a person to 

logical contact beyond the sales or service 

representative? 

III. Can your customers communicate quickly and easily 
with top management? 

IV. Do your senior executives and technical people interact 
regularly with customers? 

V. Do you use systematic, periodic surveys of customer 

satisfaction for past and present customers? 

VI. Do you have a training program on listening and customer 
contact skills for customer contact employees? 

VII. Are your customer contact employees accountable for 
customer satisfaction? 

VIII. Are customer satisfaction measures part of the 
incentive compensation plan for profit center 
managers? 


Accountability for customer satisfaction-A self-evaluation 


Service Level Agreements and The Workflow Paradigm 

A final technique for securing the customer's view on IS 
performance does not include surveys at all. Rather it involves 
a negotiated level of performance between customer and provider 
even before the product or service is delivered. This technique 
has been used effectively at the Security Pacific Automation 
company (SPAC) , the IS organization for the Security Pacific 
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Corporation, a diversified financial services corporation 
(Singleton 1988) . The SPAC program, entitled Management by 
Results (MBR) 1 , contains four components of creating and 
measuring effectiveness: strategic planning, service level 
agreements, commitment plans, and appraisal and rewards. The 
service level agreements are user-driven definition of products 
and services to provided by SPAC. These agreements contain 
measurable criteria to judge whether SPAC performance was 
excellent, above average, average, or unsatisfactory. 

The service level agreement is part of the broader concept 
of customer satisfaction described by Center and Henry (1992) . 
They propose a paradigm of evaluation that complements the 
traditional input/process/output paradigm. The new approach, 
called the Workflow Paradigm, consists of four interactions 
between the customer and the IS organization: 

Customer makes a request or provider makes an 
offer for work to be done. 

Customer and provider reach mutual agreement 
about what is to be accomplished. (This 
agreement constitutes the conditions of 
customer satisfaction.) 

Provider completes the work and reports 
completion to the customer. 

Customer assesses the work and declares 
satisfaction (or some level thereof) . 

Much like the service level agreement, the workflow builds the 
"right things" and the customer's satisfaction right into the 
definition of the work itself. The Workflow Paradigm is most 
appropriate for non-routine, less structured work where 
Input /Process/Output is more common in routine, production work. 


Opening 

Agreement 

Performance 

Acceptance 


1 " Management by Results," "MBR," and "Commitment Plan" are 
registered trademarks of Security Pacific National Bank. All 
rights reserved. 
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Company Experience 


. The growth of quality management initiatives in all facets 
of American business has driven all IS departments to assess 
themselves in some way. Common characteristics are found in IS 
departments who are most satisfied with their ability to measure 
their effectiveness. These characteristics are the following: 

IS director is a senior manager with the company. 
They function as a chief information officer and 
have comparable status to the chief financial 
officer. 

IS upper management has reguarly scheduled meetings 
with other senior managers or with a MIS operating 
committee. 

IS has a strategic plan with is well understood and 
integrated with the company's strategic plan. 

The most common approach employed to measure IS 
effectiveness is the customer survey, but this method has had 
mixed results for the companies who have used them. Surveys are 
distributed on a quarterly or annual cycle. Information returned 
from the surveys is analyzed either by one person who has been 
assigned the responsibility or by a team of people, most often 
the quality lead team. The statistical analysis goes to the IS 
department head. The IS head uses the information in direct 
reports and in initiatives that need the most dramatic change. 

Companies reported three major problems associated with 
assessment survey measurements — poor response rate, incomplete 
quantification and inconsistent customer perceptions. Response 
rates to surveys tend to be low. A 35% per cent response rate is 
considered excellent. A 22% per cent response rate is considered 
very good. Too often the response rate is so low as to render 
them virtually invalid. 

The process of surveying poses problems for the IS 
organization. Departments typically do not have experience and 
skills in survey design and analysis in-house so that the task is 
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assigned to someone who must "do their best". It is difficult to 
take open-ended comments and quantify them in a meaningful way. 
This job tends to be much larger and require more resources than 
originally allocated for the task. 

Additionally how a customer responds on a survey is easily 
influenced by their latest experience with technology. If their 
last experience was good, then they give a good report. If their 
last experience was bad, then they give negative feedback. The 
individual's opinion may change drastically during any given 
period of time. 

Making the survey results visible and publicizing their 
effects is important. Results are often kept on a LAN under 
"Performance Measures" and appear in reports. It was mentioned 
as harmful to the morale of the customer to build up the 
expectation of an assessment survey in the absence of an IS 
department's ability to use the information in a visible way. 

Another type of survey that to determine IS effectiveness is 
the "climate survey" for IS personnel themselves. Here, response 
rate is no problem and tends toward 100 per cent. The climate 
survey measures over-all job satisfaction and is given on an 18- 
month cycle. The accuracy of the survey results can be verified 
by a low personnel turn-over rate. 

The most effective way of determining custo mer sa tisfact ion 
was reported to be the executive interview. The IS department- 
placement in the organization is thought to be crucial to its 
effectiveness. The IS head who is a part of executive management 
is thought to be the most effective. One-on-one informal 
meetings with other division executives is thought to be the 
single most effective way of getting reliable feedback. 

"Quality is what the client perceives it to be . 
quality measurements may indicate a technologically- 
perfect product, but if the client is not satisfied, it 
is not a quality product." 

One of the companies interviewed was ranked by 
Comput erwor Id as one of the most effective users of IS 
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information for two years in a row. They do not use customer 
surveys nor do they rely heavily on quantitative measurements. 
They depend on feedback from business managers, such as 
evaluation of current services or requests for additional help. 
The company's IS director meets with the MIS Operating Committee 
(CEO, CFO, Sr. VP from each major business area) on a quarterly 
basis. This committee was singled out as a particularly good 
forum for two-way feedback and for seeing how IS doing in the 
field. 


At the other end of the spectrum, another company surveyed 
relies on quantitative data of IS performance and availability on 
a daily, weekly, quarterly and annual basis. This statistical 
data is used in trend analysis to determine standard and then 
compare against standard. "If we are operating above the 
standard, then we consider ourselves effective." 

Effectiveness as perceived by the stockholder was ranked 
as the hardest measurement to develop. One company reported only 
60 % satisfaction with its current effectiveness measurements, but 
it had one of the most comprehensive mission and assessments of 
the companies interviewed. They reported that the stockholder's 
opinion was one metric they wanted, but it was not being 
developed because they were not sure of the approach. For this 
company, evidence of success becomes part of their strategic plan 
which included internal customer satisfaction, employee 
satisfaction and owner (stockholder) satisfaction. 

Evidence of strategic alignment with the company is most 
often determined by the processes in place and the long-range 
planning activity. Companies who considered themselves as 
strategically aligned with the executive direction of the company 
included the concept of "custodians of corporate data" in their 
mission and roles as an IS department. 

Needed improvements that would help IS departments better 
assess their effectiveness included the following: 

improved ways of measuring enhancement activities 
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methods and measures for assessing administrative 
tasks 


— clearer identification of and measures for the 
discrete processes involved in software design, 
construction and installation 

— process which clearly links individual goals and 
measures and department goals and measures 

method for assessing stockholders expectations and 
satidfaction 


Conclusion 

The literature reveals four approaches to measuring the 
effectiveness of IS performance: 

1. Listen to the customer for the things they need 

2. Align with corporate goals 

3. Benchmark against well-respected organizations 

4. Ask yourself what critical factors lead to success 

The list of known methods for soliciting customer feedback 

are: 


Executive visit 

Survey, interview, focus group 
Complaints and compliments 
Service level agreements — --- -- 

A common set of characteristics that satisfy customers was 
identified from the literature. The list includes elements such 
as: 

Accuracy 
Timeliness 
Relevance 
Under standabi 1 ity 
Reliability 
• Completeness 
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Future Research 


Enough information was found to indicate that additional 
research in the following areas would be beneficial to ISD at 
JSC/NASA and to industry at large. Some high priority topics for 
research include the following: 

* Collect information and artifacts from industry on their 
IS performance measures and processes. 


* Develop guidelines for determining the external validity 
of the IS plan. 

* Identify checkpoints and measures for assessing the 
internal consistency of IS activities. 

* Catalog IS performance measures by industry and correlate 
perceived effectiveness with type of metric. 

* Describe the correlation between management structure and 
style with IS effectiveness. 

* Conduct a formal survey at JSC /NASA to define IS 
effectiveness . 

* Synthesize external information on IS performance 
measures with internal information and develop 
recommendations for meaningful performance measurements 
and a process for performance measurement. 
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Appendix A 




Information Systems Metrics 
SURVEY 

March 23, 1992 


Requestor: Cissy Yoes 

University of Houston — Clear Lake 

MC: 478, 2700 Bay Area Blvd. , Houston, TX 77058 

283-3327 FAX: 283-3322 

e-mail YOES@cl.uh.edu 


I am surveying members of the Information Systems Resource Center 
on their practices of measuring their performance as an 
information systems department. I am particularly interested in 
effectiveness measures. 

The information collected from the ISRC membership will be 
summarized and available to the membership on April 1, 1992. The 
summary of information collected will be included in a report on 
I/S Performance Measures curre nt ly being written for the 
Information Systems Directorate, JSC/NASA. All participants will 
receive a copy of this report which is expected to be ready for 
dissemination in May 1992. 

Individual responses to this survey will not be identified by 
company name or respondent unless explicit permission has been 
acquired to do so. 

This activity may also serve as a foundation for future research 
currently under consideration by Rudy Hirschheim, ISRC, and 
Jack Ross, Real Decision Corp. on Information systems Metrics. 

Thank you for your help! 


Respondent 


Company 


PLEASE RETURN BY Noon, Thursday, 3/26/92. 

You may FAX to 283-3322 or if you prefer to answer by telephone 
call 283-3327. Leave a message and I will call you back as soon 
as possible (or you can leave your responses). THANKS. 
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Information Systems Metrics 
Survey, 3/23/92 

RETURN to FAX 283-3322. Thanks. 


1. What is your role (function) as an I/S department? 


2. How do you assess your performance as an I/S department? 
(measurement definition, how collected, frequency) 


3. What do you do with this information? (How do you use it 
and who do you report it to?) 


4. How do you relate these to measurements of your 
effectiveness? 

Is this activity directly related to your quality 
initiatives? 


5. How satisfied are you with the way you are assessing I/S 
performance? What changes would you like to see? 


Thanks you, very much. 
Cissy Yoes, 283-3327 
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Critical Success Factors for IS Departments 


Caravella (1989) 

Formal relationship with end 
users 

Support services to meet user 
expectations 

End user's ability to manage 
the IS resource 

Employees with pride and 
satisfaction 

Timely delivery of quality, 
cost-effective information 
services 

Information assets that are 
safe, accessible, and have 
integrity 

Simple, consistent user 
interface 

Recognized by end users as vital 
and constructive contributors 


Slevin (1991) 

Network reliability 

Data communications network 

I/O services 

Documentation 

Problem reports 

Consulting 

Hardware repair 

Proactive planning 

Facilities management 

Internal support 


Raghunathan, Gupta, 

Sundar ar aghavan (1989) 

System development 

Data processing operations 

Human resource development 

Management of MIS /DP 
organization 

Support of corporate objectives 
Performance 
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Characteristics of Effective IS Products and Services 


Quality Product Characteristics 


Hiltz [1986] 

Newcomer (1991) 

Jenkins & Rickets 
(1979) 

Actual Usage 

Usefulness 


Perceived Benefit 

understandable 

Content 

Satisfaction 

current & timely 

accurate 


relevant 

relevant 


accessible 

adequate 

Davis (1989) 

adaptable 

accurate 

understandable 

Perceived Useful 

precise 

Form 

Perceived Easy to use 

valid 

timely 

(but not necessarily 


mode 

correlated) 

Ease of use 
(secondary) 

sequence 


input 

Problem-solving 

AT&T 

browse 

useful for ID 
problems 

Accurate 

Other 

useful for selecting 

Timely 

error resistant 

alternatives 

Relevant 

controlled access 

powerful modeling 

Complete 

recovery 

language 

Uniform 

time & cost saving 

flexible modeling 

Consistent 

Flexible 


language 

Understandable 


Input 

Reliable 

Ives (1983) 


understandable 

comprehensive 

documentation 

Systems stability 

Accurate 


response time 

Precise 


error proneness 

Reliable 


reliability 

Relevant 


accessibility 

Complete 

Confident in system 
Current (up-to-date) 
Timely (delivery) 


availability 
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Quality Service characteristics 


Ives (Service, 1983) 

Attitude of EDP staff 

Communication with 
staff 

CR processing 

Development time 

Relation with EDP 
staff 

Personal control 

Flexibility 

EDP Resource 
priorities 

Convenience 


Bitner (Service, 
1990) 

Reponse to delivery 
failure — extra 
compensation for 
inconvenience 

Reponse to request 
for extra service — 
go out of way to 
fulfill request 

Unsolicited employee 
action — 

doing extra things 


Ives (User 
Involvement, 1983) 

Training 

Understanding of 
system 

Participation in 
decisions 
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